Papers with auxiliary pre-
Self-Supervised Learning for Contextualized Extractive Summarization (P19-1)
Copied to clipboard
| Challenge: | Existing models for extractive summarization are usually trained from scratch with a cross-entropy loss . previous work builds an end-to-end system to learn to choose sentences without explicitly modeling document context . |
| Approach: | They propose three auxiliary pre-training tasks that learn to capture the document context in a self-supervised fashion. |
| Outcome: | The proposed models outperform existing models on a CNN/DM dataset. |
On Losses for Modern Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Devlin et al. ( 2018) released a transformer network (BERT) pre-training over two tasks: masked language modelling (MLM) and next sentence prediction (NSP). |
| Approach: | They clarify NSP's effect on BERT pre-training and explore ways to include multiple tasks into pre-train. |
| Outcome: | The proposed framework outperforms BERTBase on the GLUE benchmark using fewer than a quarter of training tokens. |